Naive Bayes and Map-Reduce

We’ll start out with a very simple learning algorithm: multinomial Naive Bayes. Our implementation is in Table 1. Each training example is a labeled document d = (i, y, (w1, . . . , wni)) with an identifier i, a label y from a small set Y = {y1, . . . , yK}, and a “bag of words”. The bag of words are wj’s, encoded here as a list of strings, so that wj is the word/token at position j of document i. For example, the bag of words for the paragraph below would be the strings: “when”, “scaling”, “to”, . . . , “large”, “datasets”, and ”:”. The test documents are the same, but their labels are unknown. Our goal is to read a training set, build a classifier, and then predict a label for each test example. When scaling to large datasets, the first thing to remember is that main memory (RAM) is, relatively speaking, limited and expensive, while disk space is less limited and cheap. So we want to carefully control how much memory we use. In particular, if you care about processing large datasets: